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Appendix A. About the study 


Stakeholders in the Commonwealth of the Northern Mariana Islands (CNMI) Public School System (PSS) and 
Northern Marianas College (NMC) are interested in examining how students’ social-emotional competencies are 
associated with their academic and behavioral outcomes. This interest stemmed from the priority assigned to 
promoting students’ social-emotional well-being and from recent research suggesting that students in the CNMI 
might not be ready for credit-bearing courses at NMC. 


Stakeholders are particularly interested in understanding high school students’ social-emotional competencies in 
relation to recent events that might have caused trauma and disrupted students’ social-emotional well-being, 
such as the COVID-19 pandemic and 2018’s super typhoon Yutu. These events have also had lasting impacts on 
infrastructure and services, and the blows to the tourism industry resulted in austerity measures that affected 
many families, communities, educators, and schools (Radio New Zealand, 2020). Understanding that exposure to 
chronic stress and trauma can interfere with students’ ability to learn (Schraml et al., 2012), PSS’s mission 
statement emphasizes providing whole-child student supports (Commonwealth of Northern Marianas Islands 
Public School System, n.d.), and PSS has also implemented a systemwide social-emotional learning initiative, 
Trauma Advised Student Advocacy: A Multi-Tiered Systems of Support Project (Project TASA). However, the 
effectiveness of these efforts is not currently being monitored through assessments of students’ social-emotional 
competencies. 


Stakeholders are also interested in identifying the social-emotional competencies that are associated with 
students’ high school and college academic and behavioral outcomes. This interest stemmed from a Regional 
Educational Laboratory (REL) Pacific study that found that many students in the CNMI appear to be unprepared 
for college when they graduate from local public high schools. The study found that 92 percent of PSS students 
who entered NMC placed into non-credit-bearing, or developmental, math courses, and 80 percent placed into 
non-credit-bearing English courses (Herman et al., 2017). Students were more likely to place into credit-bearing 
courses if they had better high school academic preparation, such as enrollment in Advanced Placement courses 
or in more advanced math courses, higher grade point averages, and higher standardized test scores. Further 
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motivated by research indicating that promoting students’ social-emotional competencies supports student 
learning and, ultimately, their academic and behavioral outcomes (Carneiro et al., 2007; Durlak et al., 2011; West 
et al., 2018), stakeholders in the CNMI expressed interest in examining the associations between students’ social- 
emotional competencies and high school and college academic and behavioral outcomes to inform measures to 
better prepare students for credit-bearing courses at NMC. 


The goal of this study was to deepen CNMI stakeholders’ understanding of students’ social-emotional 
competencies by examining student responses to a social-emotional competencies survey that PSS administered 
to students in grades 11 and 12 and any associations between survey scores and student outcomes. The survey 
assessed students’ social-emotional competencies on the basis of their aggregated scores on each of five domains: 
self-management, growth mindset, self-efficacy, sense of belonging, and social awareness. Aggregated domain 
scores were examined because this approach could guide educators in monitoring students’ social-emotional 
competencies and inform practices that promote the competencies being assessed (Taylor et al., 2018). For 
instance, understanding that students in a class or school report lower scores in one social-emotional competency 
domain can cue educators to investigate strategies that support their students in that area. Investigating the 
associations between students’ social-emotional competencies and their high school and college academic and 
behavioral outcomes can further support educators in understanding where to focus resources (West et al., 2018). 


Social-emotional competencies 

Social-emotional competencies refer to the knowledge, attitudes, and behaviors that enable students to be aware 
of their emotions, strengths, and weaknesses; manage their emotions and behaviors; build and maintain positive 
relationships; and make responsible decisions (Weissberg et al., 2015). Decades of research have shown that 
students’ social-emotional competencies play a role in promoting students’ academic performance, chances of 
graduation, and later life success (Carneiro et al., 2007; Weissberg et al., 2015). When students have the social- 
emotional competencies that enable them to manage personal challenges (such as stress) and academic 
challenges (such as studying for exams), they are better able to engage in learning (Weissberg et al., 2015). These 
patterns extend beyond the K-12 context. Research has revealed an association between students’ strong social- 
emotional competencies at age 11 and an increased likelihood of graduating high school, graduating college, and 
having stable employment (Carneiro et al., 2007). A cost-benefit analysis of programs that promote students’ 
social-emotional competencies showed that, on average across programs, every $1 spent yields an $11 return 
(Belfield et al., 2015). 


Connection between social-emotional compentencies and students’ high school and college 
outcomes 


Research has shown that practices, curricula, and programs designed to support students’ social-emotional 
competencies can promote students’ positive K-12 academic and behavioral outcomes and their chances for 
success in college and careers (Carneiro et al., 2007; Weissberg et al., 2015). Academic outcomes include grade 
point average and standardized test scores. Behavioral outcomes include conduct and attendance. Although 
research suggests that practices designed to promote students’ social-emotional competencies also promote 
students’ positive academic and behavioral outcomes, less research has examined the extent to which specific 
social-emotional competencies are more (or less) strongly associated with students’ academic and behavioral 
outcomes (Crowder et al., 2020). One report hypothesized that social-emotional competencies that relate to 
student behaviors, such as self-management, would be more strongly associated with these outcomes than social- 
emotional competencies that relate to knowledge and attitudes, such as social awareness (Farrington et al., 2012). 
This is because behaviors are observable and demonstrate the extent to which students are engaged in their 
schoolwork. For instance, students who turn in their work on time are more likely to have better outcomes than 
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students who do not. Although social-emotional competencies that relate to knowledge and attitudes play a role, 
the authors hypothesized that their association with student outcomes occurs by promoting positive behaviors. 
These hypotheses were supported by a study using data from the California Office to Reform Education (CORE) 
District survey, which showed that self-management and self-efficacy were more strongly correlated with high 
school students’ grade point average than were growth mindset and social awareness (West et al., 2018). Self- 
management and growth mindset were most strongly associated with high school students’ math and English 
language arts test scores. 


It is important to note, however, that most of these findings reflect student populations within the continental 
United States. There has been limited research on whether the same patterns of associations hold in different 
cultural contexts. 


Social-emotional competencies in the Commonwealth of the Northern Mariana Islands 


To support efforts to more deeply understand high school students’ social-emotional competencies and how 
they relate to students’ high school and college academic and behavioral outcomes, the CNMI PSS administered 
a social-emotional competency survey to grade 11 and 12 students that assessed five social-emotional 
competency domains during the 2018/19 academic year. 


Scales for four competency domains (growth mindset, self-management, self-efficacy, and social awareness) were 
obtained from the CORE Districts Social and Emotional Learning Survey (Transforming Education, 2016).* Growth 
mindset refers to a belief in one’s ability to learn. Self-management refers to the ability to regulate one’s emotions, 
thoughts, and behaviors. Self-efficacy refers to a belief in one’s ability to reach their academic goals. Social 
awareness refers to the ability to take another person’s perspective and to understand social norms for behavior. 
A study examining the reliability and predictive validity of these four scales, using a sample of 378,465 students 
across five school districts, found that all four scales had adequate internal reliability (with Cronbach’s alphas 
above .70) and that elementary, middle, and high school students’ scores for all four scales were positively 
correlated with grade point average, math standardized test scores, and English language arts standardized test 
scores (West et al., 2018). Further, another study examining the construct validity of the four scales using 
exploratory factor analyses verified the expected four-dimensional structure (Meyer et al., 2018). 


In addition to these four scales, the PSS survey included the Simple School Belonging Scale, which assessed the 
extent to which students believed that others in their school community cared about and accepted them (Whiting 
et al., 2017). Whiting et al. (2017) provided evidence for construct validity, as the questions within the scale 
supported a unidimensional structure (they measured one construct) as well as internal reliability (Cronbach’s 
alpha of .91). 
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Appendix B. Methods 


This appendix provides further details on the study’s data sources, samples, missing data, measures, variables, 
and analytical approach. 


Data sources 


The study examined two types of data: survey data from the Commonwealth of the Northern Mariana Islands 
(CNMI) Public School System (PSS) and administrative data from PSS and Northern Marianas College (NMC). 


Survey data. In May 2019 students in grades 11 and 12 who were enrolled in PSS high schools completed a brief 
survey about their self-management skills, growth mindset, sense of self-efficacy, sense of belonging, and social 
awareness. Students were informed that the survey was confidential and voluntary. Students received paper 
copies of the survey and were asked to write their student identification number at the top of their survey form. 
Scanned and printed copies of the completed surveys were provided to the Regional Educational Laboratory 
Pacific, whose staff entered the data into an electronic format. 


Administrative data. The CNMI PSS provided student demographic data, such as gender, ethnicity, and grade level, 
as well as students’ academic outcome data (grade point average and standardized test scores) and behavioral 
outcome data (attendance). NMC provided data on college course enrollment, number of credits attempted and 
earned, first semester grade point averages, and persistence into a second semester. 


Data processing and creation of analytic samples 


A total of 890 high school students (438 in grade 11, 276 in grade 12, and 164 in another grade or with no reported 
grade’) from four of the five PSS high schools and from one alternative high school completed the survey (figure 
B1).3 Of these students, 553 (62 percent) provided their student identification number. PSS relies on three 
separate identification systems. The study team first linked the IDs given by the students to the “Admin Plus” 
identification number (APID) in the PSS dataset, because PSS reported that most students would use this number. 
Next, unmatched IDs were linked using a unique identification number assigned by PSS. In total, 439 students’ (49 
percent) were linked.° 


After linking students’ survey responses to their PSS administrative data, the study team relied on a similar 
procedure for identifying the students who were in grade 12 when they completed the survey (May 2019) and 
who enrolled in NMC in the fall 2019 semester. Because some student names were not identical across the two 
datasets (due to misspellings or slight name changes from year to year) and there was no shared ID between high 
school and NMC, inexact matches that met a threshold of similarity were matched using fuzzyjoin (Robinson, 
2019), an R package (R Core Team, 2020), to create a list of matched students based on first and last name and 


? These 164 students were excluded from the analyses. 

3 The four PSS high schools that participated in this study were Kagman High School, Marianas High School, Saipan Southern High School, 
and Tinian Jr./Sr. High School. The alternative high school that participated in this survey was Da’Ok Academy. The survey was not 
administered at Dr. Rita H. Inos High School on Rota. 

4 Three students were removed from the analysis because the IDs given linked to different students at the same school depending on 
whether the Unique ID or APID was used and there was no other disambiguating information. 

> Based on conversations with stakeholders, the study team speculates that this might have been the first time that the PSS had 
administered a survey that was not anonymous. As such, it is likely that protocols and procedures for communicating about or administering 
a confidential survey were not already established. The low number of linkable surveys might have stemmed from confusion about the 
purpose of the study and the need to provide student identification numbers. Relatedly, stakeholders indicated that students might not 
have known their identification numbers; it would have been the schools’ responsibility to provide the identification numbers to students, 
which may not have occurred. 
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date of birth, using a probability of likely matches. The study team examined this list and compared students’ 
names, gender, and date of birth across the joined data sources to confirm that matches were correct. Incorrect 
matches were removed, and the original unmatched pair of students was again considered for matching with 
other records from the opposite database. This resulted in identifying 82 students who transitioned from PSS to 
NMC for the fall 2019 semester. The sample represents 35 percent of students from these CNMI public high 
schools who graduated in 2019 and who enrolled at NMC in fall 2019. 


Figure B1. Number of students in each step of the determination of the analytic samples 


Students who completed the survey 


(n = 890) 
Students who completed the survey and Students with Commonwealth of the Northern Mariana 
listed a student identification number Islands Public School System administrative data 
(n = 553) (n = 1,371) 
Students who completed the survey and linked 
with Commonwealth of the Northern Mariana Students with Northern Marianas 
Islands Public School System administrative data College administrative data 
[Sample for research questions 1 and 2] (n = 1,252) 
(n = 439) 


= Lo 


Students who completed survey and linked 
with both Commonwealth of the Northern 
Mariana Islands Public School System 
administrative data and Northern Marianas 
College administrative data 
[Sample for research questions 3 and 4] 
(n = 82) 


Note: The number of students who had data for each outcome varied. Students without data for an outcome in question were dropped from that analysis. 
For research question 4 two students had missing predictor values and were dropped from the regression models. 

Source: Authors’ compilation based on linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands 
Public School System and Northern Marianas College for students who were in one of the five participating high schools in 2018/19. 


The study team created two analytic samples to answer the research questions. The first analytic sample included 
the 439 grade 11 and 12 students for whom both survey and PSS administrative data were available. This sample 
represented 32.0 percent (n = 1,371) of grade 11 and 12 students enrolled in all CNMI public high schools at the 
time the survey was administered and 33.8 percent (n = 1,299) of grade 11 and 12 students attending a school 
where the survey was administered at the time it was administered. This sample was used to answer research 
questions 1 and 2. The second analytic sample included the 82 students who completed both the social-emotional 
competencies survey and for whom linked PSS and NMC administrative data were available. This sample, which 
included only grade 12 students, was used to answer research questions 3 and 4. 


Students’ social-emotional competencies 


The survey of students’ social-emotional competencies consisted of five scales. Four scales were derived from the 
California Office to Reform Education (CORE) Districts Social and Emotional Learning Survey: self-management, 
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growth mindset, self-efficacy, and social awareness (Transforming Education, 2016). The fifth scale, sense of 
belonging, was derived from the Simple School Belonging Scale and measured students’ sense of belonging at 
school (Whiting, et al., 2017). Each of the five scales, or social-emotional competency domains (tables B1 and B2), 
is described here: 


Self-management. Consisted of nine items that asked students to answer how often they engaged in a series 
of behaviors during the past 30 days. Students used a Likert scale ranging from 1 (almost never) to 5 (almost 
all the time) for their responses. An example is “I got my work done right away instead of waiting until the last 
minute.” 


Growth mindset. Consisted of four items that asked students to indicate how true each statement was for 
them using a Likert scale ranging from 1 (not at all true) to 5 (completely true). An example is “My intelligence 
is something that | can’t change very much.” Responses were reverse scored before they were aggregated so 
that higher values represent greater growth mindset. 


Self-efficacy. Consisted of four items that asked students to indicate how confident they are in achieving 
positive outcomes in their classes. Students used a Likert scale ranging from a 1 (not at all confident) to 5 
(completely confident) in their responses. An example is “I can earn an A in my classes.” 


Sense of belonging. Consisted of 10 items that asked students to rate the extent to which they agreed or 
disagreed with statements such as “Students at my school are friendly to me.” Due to a clerical error, the 
response bubbles signaling students to select a response option were omitted for one social awareness item, 
“Other students here like me the way | am.” As a result, only 203 students responded to this question. To 
ensure that sense of belonging scores reflected the same construct for all students, this item was omitted 
when calculating students’ average sense of belonging scores. The results using this scale were compared to 
one that included the omitted item, and there were no substantive differences.® 


Social awareness. Consisted of eight items that asked participants to indicate how often they engaged in a 
series of behaviors during the past 30 days. Response options varied depending on the item. For example, the 
question “How carefully did you listen to other people’s points of view?” was anchored on a Likert scale 
ranging from 1 (not carefully at all) to 5 (extremely carefully). 


Responses to the items within each of the five scales were averaged to create five separate scores (one for each 


social-emotional competency domain) for each student, with higher scores reflecting greater social-emotional 
competency. The possible values ranged from 1 to 5 for each of the scores (see tables C1 and C3 in appendix C for 
descriptive results of the social-emotional competency scores). All five scales demonstrated adequate internal 
reliability across both samples (table B1), with Cronbach’s alpha values greater than .70, which, according to 
DeVellis (2003), indicate adequate internal reliability. 


5 When item 5 is added to the sense of belonging scale, the Cronbach’s alpha is 0.83 for the high school sample and 0.85 for the college 
sample. When the item is removed, it is 0.80 for the high school sample and 0.83 for the college sample. 
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Table B1. Internal reliability estimates for students in the Commonwealth of the Northern Mariana Islands 
Public School System and Northern Marianas College, by social emotional competency domain, 2019 
(Cronbach's alpha) 


Student sample 


Public School System Northern Marianas College 
NYofoir-]ex-anend (el are] mee) aay ol=1n=t aon] (eis) (n = 82) 
Self-management 79 74 
Growth mindset 74 76 
Self-efficacy 89 .90 
Sense of belonging? .80 83 
Social awareness .80 .78 


Note: Sample includes grade 11 and 12 high school students in five participating high schools in spring 2019 and a subsample of those students who attended 
Northern Marianas College in fall 2019. 

a. Excludes responses to one social awareness item (“Other students here like me the way | am”) that, because of a clerical error, was missing its response 
bubbles. 

Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System and Northern Marianas College. 


Table B2. Alignment of self-report survey questions with social-emotional competency scales, May 2019 


Social-emotional competency and corresponding survey items 
Self-management 

. [came to class prepared. 

. |remembered and followed directions. 

. [got my work done right away instead of waiting until the last minute. 


. | paid attention, even when there were distractions. 


1 
2 
3 
4 
5. | worked independently with focus. 
6. | stayed calm even when others bothered or criticized me. 
7. | allowed others to speak without interruption. 
8. | was polite to adults and peers. 
9.1 kept my temper in check. 
Growth mindset 
10. My intelligence is something that | can't change very much. 
11. Challenging myself won't make me any smarter. 
12. There are some things | am not capable of learning. 
13. If | am not naturally smart in a subject, | will never do well in it. 
Self-efficacy 
14. | can earn an A in my classes. 


15. | can do well on all my tests, even when they’re difficult. 


16. | can master the hardest topics in my classes. 


17. | can meet all the learning goals my teachers set. 


Sense of belonging 

18. Students at my school notice when | am good at something. 

19. Other students at my school take my opinions seriously. 

20. Students at my school are friendly to me. 

21. | am included in lots of activities at my school. 

22. Other students here like me the way | am. 

23. | like to think of myself as similar to others at my school. 

24. People at my school care if | am absent. 

25. | feel like my ideas count at my school 

26. | feel like | matter to students at my school. 

27. Students really listen to me when | am at school. 

Social awareness 

During the past 30 days... 

28. How carefully did you listen to other people’s points of view? 

29. How much did you care about other people’s feelings? 

30. How often did you compliment others’ accomplishments? 

31. How well did you get along with students who are different from you? 

32. How clearly were you able to describe your feelings? 

33. When others disagreed with you, how respectful were you of their views? 
34. To what extent were you able to stand up for yourself without putting others down? 
35. To what extent were you able to disagree with others without starting an argument? 


Source: Survey constructed by the Commonwealth of the Northern Mariana Islands Public School System based on the California Office to Reform Education 
District Social and Emotional Learning Survey (Transforming Education, 2016) and the Simple School Belonging Scale (Whiting, et al., 2017). 


High school student characteristics 

For research questions 1 and 2 the study team examined the associations between students’ self-reported social- 
emotional competencies and their high school academic and behavioral outcomes while controlling for high 
school student characteristics, as reported in PSS administrative data. The study included the following high school 
student characteristics (table B3): 


e Gender. The student’s reported gender, coded as female (coded as 0) or male (coded as 1), with female as the 
reference group.’ 


e Ethnicity. The student’s reported ethnicity, coded as Pacific Islander/Native Hawaiian (coded as 3), Asian 
(coded as 2), or other (coded as 1). The reference group was Pacific Islander/Native Hawaiian. 


e Grade level. The grade level of the student, coded as grade 11 (coded as 0) or grade 12 (coded as 1), with 
grade 11 as the reference group. 


e High school. High school in which the student was enrolled during 2018/19 school year. Given the small 
number of students for each school, this variable was coded as Marianas High School (coded as 1) or other 
PSS school (for the other four schools, coded as 0), with other PSS school as the reference group. 


7 The reference group refers to the values represented by the constant in the regression analyses. 
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Table B3. High school student characteristics, 2018/19 


Student characteristic Percent of students 


Gender 

Male 54.2 
Female? 45.8 
Ethnicity 

Pacific Islander/Native Hawaiian? 41.0 
Asian 55.1 
Other 3.9 
Grade level 

Grade 11? 54.7 
Grade 12 45.3 
High school’ 

Da’Ok Academy 1.4 
Kagman High School 0.4 
Marianas High School 88.4 
Saipan Southern High School 3.6 
Tinian High School 6.2 


Note: Percentages might not sum to 100 because of rounding. 

a. Reference group. 

b. The four high schools excluding Marianas High School were the reference group. 

Total number of students in the final sample is 439. 

Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System. 


College student characteristics 


For research questions 3 and 4 the study team examined the associations between students’ self-reported social- 
emotional competencies on the May 2019 survey and their college academic and behavioral outcomes while 
controlling for college student characteristics. The study team examined the following college student 
characteristics, as reported in PSS and NMC administrative data (table B4): 


e Gender. The reported gender of the student, coded as female (coded as 0) or male (coded as 1), with female 
as the reference group. 


Northern Marianas descent. Whether the student was of Northern Marianas descent, defined as being 
CHamoru or Refaluwasch (Carolinian), coded as 0 (no) or 1 (yes). The reference group was not of Northern 
Marianas descent. 


e Pell Grant recipient. Whether a student received a Pell Grant during their first semester at NMC, coded as 0 
(no) or 1 (yes). The reference group was not a Pell Grant recipient. 
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Table B4. College student characteristics, 2019 


Percent of students 


Student characteristic (n = 82) 
Gender 

Male 36.6 
Female? 63.4 


Northern Marianas descent 

Yes 57.3 
No? 40.2 
Pell Grant recipient 

Yes 74.4 
No? 23.2 


Note: Percentages may not sum to 100 because of rounding and missing data. 
a. Reference group. 
Source: Authors’ analysis of linked May 2019 survey data and administrative data from Northern Marianas College. 


High school academic outcomes 
For research questions 1 and 2 the study team examined the following high school academic outcome variables: 
e Grade point average. A student’s cumulative high school grade point average at the end of the 2018/19 school 


year. Values for this variable were continuous, with a possible range of 0 to 100 and with 0-62 corresponding 
to an F average, 63-72 to a D average, 73-82 to a C average, 83—92 to a B average, and 93-100 to an A average. 


Grade 10 ACT Aspire math scale score. A student’s grade 10 ACT Aspire math scale score, with a possible range 
of 400 to 460; a score of 432 is the ACT Aspire grade 10 benchmark. 


Grade 10 ACT Aspire reading scale score. A student’s grade 10 ACT Aspire reading scale score, with a possible 
range of 400 to 442; a score of 428 is the ACT Aspire grade 10 benchmark. 


High school behavioral outcome 
For research questions 1 and 2 the study team examined the following high school behavioral outcome variable: 
e Number of days absent from at least one course. A student who missed at least one period in a day during the 


2018/19 school year was considered absent for the day.® 


College academic outcomes 
For research questions 3 and 4 the study team examined the following college academic outcome variables: 


e First semester college grade point average. A student’s college grade point average for the first semester at 
NMC. Values for this variable were continuous, with a possible range of 0 to 4. 


e = Enrollment in developmental math. Whether a student enrolled in a developmental math course (coded as 1) 
or credit-bearing math course (coded as 0) at NMC. 


e Enrollment in developmental English. Whether a student enrolled in a developmental English course (coded 
as 1) or credit-bearing math course (coded as 0) at NMC. 


8 Absences included both excused and unexcused absences; however, this variable excluded students who arrived late or were suspended 
and students attending school-sponsored events and trips. 
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e Completed all attempted credits. Whether a student completed all attempted credits during the first semester 
at NMC (coded as 1) or not (0). 


College behavioral outcome 


To answer research questions 3 and 4, the study team examined the following college behavioral outcome 
variable: 


e Persistence into second semester. Whether the student persisted into a second semester at NMC (coded as 1) 
or not (coded as 0). 


Missing data 

The study’s sample of 439 students for research questions 1 and 2 represented 49.3 percent of grade 11 and 12 
students who took the survey and 32.0 percent of the grade 11 and 12 public school population (see appendix D 
for nonresponse analyses). Of the 439 students included in the final analytic sample for research questions 1 and 
2, 13 students were missing data for two social-emotional competencies (table B5), and 59 students were missing 
data for at least one outcome variable. Students who were missing data for an outcome variable were dropped in 
regression models for that outcome, so the number of cases for each regression model for research question 2 
varied across outcomes: n = 439 students for number of days absent from at least one course, n = 433 for high 
school grade point average, n = 383 for grade 10 ACT Aspire reading scale score, and n = 380 for ACT Aspire math 
scale score. 


Table B5. Percentage of students missing data for variables in the regression models for research question 2 


Percent of students 


Outcome variable with missing data 
Self-efficacy 2.96 
Sense of belonging 3.42 
High school grade point average 1.37 
Grade 10 ACT Aspire math scale score 13.44 
Grade 10 ACT Aspire reading scale score 12.76 

Note: n= 439. 


Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and 12 students in one of five participating high schools in 2018/19. 


To help preserve the study’s sample size and to include students with missing data for predictor variables in the 
study, the study team conducted multiple imputation using the Multivariate Imputation by Chained Equations 
package (van Buuren & Groothuis-Oudshoorn, 2011) in R (R Core Team, 2020). Multivariate Imputation by Chained 
Equations is appropriate for imputing continuous, binary, and ordered and unordered categorical data. The 
imputation models included high school student characteristics and the outcome variables examined in this study. 
The imputation procedures created five versions of complete data sets by using existing values for variables to 
predict missing variables. Analyses were then performed across datasets, and the results were pooled to yield one 
set of results for each analysis. Findings from these analyses are presented in the main report. To determine 
whether the study findings differed based on methods used to address missing data, the study team also 
conducted sensitivity analyses using the sample of students who had complete data. The findings from those 
analyses are presented in appendix D. 


For research question 4, 2 of 82 students in the sample were missing at least one predictor variable value. Given 
the small number of students with missing data, listwise deletion was used in the regression models. In addition, 
not all students had data for each of the outcome variables used for research questions 3 and 4. For first semester 
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college grade point average and persistence into a second semester, outcome data were missing for 2 students 
(n = 80), while for enrollment in developmental math and enrollment in developmental English outcome data 
were missing for 28 students (n = 54).° All students had data for completing all attempted credits. Students with 
missing data for an outcome variable were dropped in the analyses of that variable. 


Analysis methods 


This section describes the analysis methods used to answer the research questions. 


Descriptive analyses. For research question 1 the study team used R (R Core Team, 2020) to calculate means to 
describe the social-emotional competencies of grade 11 and 12 students who were enrolled in one of the five 
participating CNMI public high schools during the 2018/19 school year. 


For research question 3 the study team used R (R Core Team, 2020) to calculate means to describe the social- 
emotional competencies of students who graduated from a CNMI public high school in spring 2019 and then 
enrolled as first-time students at NMC during the fall 2019 semester. 


The variables included in these descriptive analyses are described above. 


Multiple regression analyses. For research question 2 the study team used R (R Core Team, 2020) to run four 
multiple regression models. Three of the models used linear regression, because the outcome variables high 
school grade point average, grade 10 ACT Aspire math scale scores, and grade 10 ACT Aspire reading scale scores 
were continuous values. The model for the outcome variable number of days absent from at least one course 
used a negative binomial regression model, because the outcome variable is a count variable with observed 
overdispersion (variance is greater than the mean). 


The regression models for research question 2 included student gender, ethnicity, grade level, high school in which 
the student was enrolled, and average scores on each of the five social-emotional competency domains as 
predictor variables. 


The continuous social-emotional competency predictors did not have a meaningful value of 0. Therefore, each 
continuous predictor was centered around its mean in preparation for the multiple regression analyses. The 
categorical predictors were dummy coded, and the reference groups were female students, Pacific 
Islander/Native Hawaiian students, grade 11 students, and high schools other than Marianas High School. 


For research question 4 the study team used R (R Core Team, 2020) to run five multiple regression models. One 
of the models used linear regression, because the outcome variable first semester college grade point average is 
a continuous value. The model for the outcome variables enrollment in developmental math, enrollment in 
developmental English, completed all attempted credits, and persistence into a second semester used logistic 
regression because the outcome variables are dichotomous. 


The regression models for research question 4 included students’ gender, Northern Marianas descent, Pell Grant 
recipient status, and average score on the five social-emotional competence domains as predictor variables. 


The continuous social-emotional competency predictors did not have a meaningful value of 0. Therefore, each 
continuous predictor was centered around its mean in preparation for the multiple regression analyses. The 
categorical predictors were dummy coded, and the reference groups were female students, not of Northern 
Marianas descent, and not a Pell Grant recipient. 


° The number of students with missing data for the outcome variables enrollment in developmental math and enrollment in developmental 
English is higher than for the other outcome variables because not all students enrolled in a math or English course during their first 
semester at Northern Marianas College. Course placement data were unavailable to the study team, so course enrollment data were used. 
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The linear regression models used for research questions 2 and 4 for predicting the value of the continuous 
outcome variables, given the predictors described above, are summarized as follows: 


Yi = Bo + B1X1i + BoX2i + B3X3i + BeXei + & 


where Y, is the value of the outcome variable for student i, Bo is the estimated value of the outcome of interest 
when all predictor variables take on the value of 0, Bi, are the coefficients for capturing the associations between 
the predictor variables and the value of meeting the outcome of interest, Xi;-«; are the values of the predictor 
variables for student /, and ¢; is the difference between the predicted value of Y and the actual value of Y for 
student i. 


The negative binomial regression model used for research question 2 for predicting the number of days a student 
was absent in at least one course is summarized as follows: 


In(¥;) = Bo + BiX1; + B2X2i + B3X3i + BeX Ki + Ei; 


where In(¥;) is the predicted log count of the number of days a student was absent from at least one course for 
student ij, Bo is the estimated log count of number of days a student was absent from at least one course when all 
predictor variables take on the value of 0, Bi is the coefficient for capturing the associations between the 
predictor variables and the log count of the number of days a student was absent from at least one course, X4;-; 
is the value of the predictor variables for student /, and € is the difference between the predicted value of In(¥;) 
and the actual value of In(¥Y;) for student i. 


The logistic regression models for research question 4 described above are summarized as follows: 


In F os = Bo + BiX4; + B2X2i + B3X3i «+ Bu Xe + & 
U 


where P, is the predicted probability of student i achieving the outcome of interest, Bo is the estimated log odds 
of meeting the outcome of interest when the predictor variables take on the value of 0, Bi is the coefficient for 
capturing the associations between the predictor variables and log odds of meeting the outcome of interest, 
X1-« is the value of the predictor variables for student i, and ¢€; is the difference between the predicted value of P, 
and the actual value of P; for student i. 


For each regression model for research questions 2 and 4, all the predictor variables were entered simultaneously. 
After running these analyses, the study team examined the predictor variables for evidence of multicollinearity. 
Multicollinearity occurs when there are high correlations among predictor variables, which can negatively affect 
the results of the analysis. A common way to detect multicollinearity is to examine the variance inflation factors 
of predictor variables. A variance inflation factor estimates the extent to which a regression coefficient’s variance 
is inflated because of multicollinearity among predictor variables. Variance inflation factors that are greater than 
5 suggest high multicollinearity. The variance inflation factors were below 5 for each model, indicating low 
multicollinearity. After running the logistic and negative binomial regression models, the study team calculated 
odds ratios for each predictor variable. 


Finally, the study team used the emmeans package (Lenth, 2019) in the R program (R Core Team, 2020) to calculate 
the predicted means (for linear and negative binomial regression models) for each outcome variable by the social- 
emotional competency predictor variables. The predicted means reflect the predicted average values of the 
outcome variable at specified values of a predictor variable, while the model averages across the levels of other 
categorical predictor variables and while other continuous variables are held at their average values. The study 
team calculated predicted means at values of 1.0, 2.0, 3.0, 4.0, and 5.0 for each social-emotional competency 
predictor variable (see appendix C). 
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Appendix C. Supporting analysis 


This appendix provides detailed results for the descriptive and regression analyses discussed in the main report. 


Detailed results of descriptive analyses 


This section describes the percentages, means, medians, standard deviations, minimum values, and maximum 
values of the final analytic samples for the examination of the social-emotional competencies and academic and 
behavioral outcomes (research questions 1 and 3; tables C1-C5). 


Table C1. Descriptive statistics for social-emotional competencies for high school students (research 
question 1) 


Syt-larel-lae| 
AYofolr-]ex-anea(ear-] mee) aalol-inclalemeleleat-lin Mean Median deviation Minimum Maximum 
Self-management 4.02 4.00 0.54 1.22 5.00 
Growth mindset 3.91 4.00 0.85 1.25 5.00 
Self-efficacy 3.16 3.00 0.98 1.00 5.00 
Sense of belonging 3.27 3.22 0.60 1.44 5.00 
Social awareness 3.74 3.75 0.58 1.00 5.00 


Note: n= 439. 
Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and 12 students in one of the five participating high schools in 2018/19. 


Table C2. Descriptive statistics for high school academic and behavioral outcomes (research question 1) 


Sir-Talerlae| 
High school outcome variable Mean Median deviation Minimum Maximum 
High school grade point average 83.24 84.03 9.56 37.00 99.41 
Grade 10 ACT Aspire math scale score 418.91 418.00 6.52 405.00 444.00 
Grade 10 ACT Aspire reading scale score 418.26 417.00 6.82 403.00 435.00 
Number of days absent from at least one course 17.19 12.00 18.74 0.00 120.00 
Note: n= 439. 


Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and 12 students in one of the five participating high schools in 2018/19. 


Table C3. Descriptive statistics for social-emotional competencies for high school students who went on to 
attend Northern Marianas College (research question 3) 


St-larel-lae| 
NYofeir-]ex-danea(elar] mexelpalotqinclaremelelear-lin Mean Median deviation Minimum Maximum 
Self-management 4.16 4.22 0.56 1.22 5.00 
Growth mindset 4.10 4.25 0.82 1.50 5.00 
Self-efficacy 3.31 3.25 0.97 1.00 5.00 
Sense of belonging 3.45 3.44 0.62 1.89 5.00 
Social awareness 3.80 3.88 0.59 2.12 5.00 


Note: n= 82. 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System and Northern Marianas College for first time first-year students at Northern Marianas College during fall 2019 who graduated from one of five 
participating high school in spring 2019. 


Table C4. Descriptive statistics for college academic outcome variable: First semester grade point average 
(research question 3) 


Sirlaler-lae| 
College outcome variable Mean WKexe freTa) deviation Minimum WED ATenlelan) 
First semester grade point average 2.83 3.00 1.17 0.00 4.00 


Note: n= 82. 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System and Northern Marianas College for first time first-year students at Northern Marianas College during fall 2019 who graduated from one of five 
participating high schools in spring 2019. 


Table C5. Number and percentage of students with other college academic and behavioral outcome data 
(research question 3) 


College outcome variable Number Percent 
Enrollment in developmental math 44 81.48 
Enrollment in developmental English 23 42.59 
Completed all attempted credits 62 75.61 
Persistence into a second semester 68 85.00 


Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System and Northern Marianas College for first time first-year students at Northern Marianas College during fall 2019 who graduated from one of five 
participating high schools in spring 2019. 


Detailed results from regression analyses 


This section provides the detailed results of the regression analyses examining high school academic and 
behavioral outcomes (tables C6—C10) and college academic and behavioral outcomes (tables C11—C16) used to 
answer research questions 2 and 4. Where appropriate, each table provides the coefficient and the corresponding 
p-value for a given social-emotional competency domain. For linear regression models the coefficient (f) 
represents the estimated change in the outcome variable given a one-unit change in the predictor variable. For 
negative binomial regression models the coefficient represents the estimated change in log counts of the outcome 
variable given a one-unit change in the predictor variable. For logistic regression models the coefficient represents 
the estimated change in odds ratio given a one-unit change in the predictor variable (see appendix B for additional 
details about the regression analyses and corresponding coefficients). In the tables the coefficient, log count, odds 
ratio, and corresponding p-value are bolded and italicized when the association between that domain and the 
outcome variable is statistically significant (p < .05). Included in these detailed results are the predicted means for 
the social-emotional competency domains that are significant predictors of at least one of the outcomes examined 
(tables C10 and C16). 
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Table C6. Results of regression model predicting high school grade point average, 2019 


High school grade point average 


Predictor variable p-value 


Student background characteristic 


Grade, 12 0.35 .646 
Gender, male -4,47 <.001 
Ethnicity, Asian 5.02 <.001 
Ethnicity, other 5.49 .010 
High school, Marianas High School -4,72 <.001 


Social-emotional competency domain 


Self-management 1.46 .080 
Growth mindset 1.57 .001 
Self-efficacy 3.66 <.001 
Sense of belonging 0.29 .685 
Social awareness -1.98 .010 
Constant 86.71 <.001 


Note: n = 433. Coefficients and p-values shown in bold italic indicate that a given domain is a statistically significant predictor (p < .05) of the outcome. For 
student characteristics the reference groups were grade 11 students at high schools other than Marianas High School, female, Pacific Islander/Native 
Hawaiian, and students who had the average score for each of the social-emotional competency domains. 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and grade 12 students in one of the five participating high schools in 2018/19. 


Table C7. Results of regression model predicting grade 10 ACT Aspire math scale scores 
Grade 10 ACT Aspire math scale scores 


Predictor variable p-value 


Student background characteristic 


Grade, 12 1.36 .030 
Gender, male -1.47 .017 
Ethnicity, Asian -1.79 .005 
Ethnicity, other 3.31 .070 
High school, Marianas High School —1.03 311 
Social-emotional competency domain 

Self-management -0.17 .800 
Growth mindset 1.22 .002 
Self-efficacy 2.27 <.001 
Sense of belonging 0.63 .300 
Social awareness —0.63 .309 
Constant 418.91 <.001 


Note: n = 380. Coefficients and p-values shown in bold italic indicate that a given domain is a statistically significant predictor (p < .05) of the outcome. For 
student characteristics the reference groups were grade 11 students at high schools other than Marianas High School, female, Pacific Islander/Native 
Hawaiian, and students who had the average score for each of the social-emotional competency domains. 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and grade 12 students in one of the five participating high schools in 2018/19. 


Table C8. Results of regression model predicting grade 10 ACT Aspire reading scale scores 


Grade 10 ACT Aspire reading scale scores 


Predictor variable p-value 


Student background characteristic 


Grade, 12 1.13 .083 
Gender, male -3.04 <.001 
Ethnicity, Asian 1.44 .031 
Ethnicity, other 2.34 .186 
High school, Marianas High School -1.28 .224 
Social-emotional competency domain 

Self-management 0.67 341 
Growth mindset 1.61 <.001 
Self-efficacy 1.75 <.001 
Sense of belonging -0.44 483 
Social awareness —1.14 .078 
Constant 419.63 <.001 


Note: n = 383. Coefficients and p-values shown in bold italic indicate that a given domain is a statistically significant predictor (p < .05) of the outcome. For 
student characteristics the reference groups were grade 11 students at high schools other than Marianas High School, female, Pacific Islander/Native 
Hawaiian, and students who had the average score for each of the social-emotional competency domains. 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and grade 12 students in one of the five participating high schools in 2018/19. 


Table C9. Results of negative binomial regression model predicting number of days absent from at least one 
high school course 


Number of days absent from at least one course 


Predictor variable Log count p-value 


Student background characteristic 


Grade, 12 -0.29 .002 
Gender, male 0.15 121 
Ethnicity, Asian 0.24 .013 
Ethnicity, other —0.26 311 
High school, Marianas High School —0.34 .045 


Social-emotional competency domain 


Self-management -0.17 .091 
Growth mindset —0.02 .700 
Self-efficacy -0.24 <.001 
Sense of belonging 0.05 .560 
Social awareness 0.10 .274 
Constant 3.29 <.001 


Note: n = 439. Log counts and p-values shown in bold italic indicate that a given domain is a statistically significant predictor (p < .05) of the outcome. For 
student characteristics the reference groups were grade 11 students at high schools other than Marianas High School, female, Pacific Islander/Native 
Hawaiian, and students who had the average score for each of the social-emotional competency domains. 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and grade 12 students in one of the five participating high schools in 2018/19. 


Table C10. Predicted means for each high school outcome, by social-emotional competency response category 


eTg-Jo[=) (eT g-\e (=m 0) Grade 10 Number of days 
Yolo] anon lover] point ACT Aspire math ACT Aspire reading absent from at least 


(oko lan) ol=1n-darea vane (elaar-liam-lare! average LXer] [-mcvere) a=) KYor-] (mere) a=) one course 
response category (eo) (aieet=10)) (eet:i)) (ei) 


Growth mindset 


1 81.4 416.7 414.8 ’ 

2 83.0 417.9 416.3 $ 

3 84.6 419.1 417.8 2 

4 86.2 420.2 419.4 : 

5 87.8 421.4 420.9 ° 
Self-efficacy 

1 77.9 415.2 415.3 30.23 
2 81.7 417.4 417.1 23.81 
3 85.4 419.7 418.9 18.54 
4 89.2 422.0 420.7 14.59 
5 93.0 424.3 422.5 11.47 
Social awareness 

1 92.1 7 4 a 

2 89.9 _ ? 7 

3 87.7 2 a ? 

4 85.5 si a : 

5 83.2 : a : 


Note: Survey items were rated on a 1-5 Likert scale, with higher scores reflecting greater social-emotional competency. The predicted marginal means are 
calculated from the regression models (see appendix B for how the predicted means were calculated). 

a. The finding was nonsignificant; only values for significant findings are included (see tables C6—C9). 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and grade 12 students in one of the five participating high schools in 2018/19. 


Table C11. Results of regression model predicting first semester grade point average at Northern Marianas 
College 


First semester grade point average 


Predictor variable p-value 


Student background characteristic 


Pell Grant recipient -0.39 .212 
Gender, male -0.32 .261 
Northern Marianas descent -0.22 429 


Social-emotional competency domain 


Self-management 0.21 427 
Growth mindset 0.33 .049 
Self-efficacy 0.11 464 
Sense of belonging —0.07 .780 
Social awareness —0.41 .106 
Constant 3.33 <.001 


Note: n = 80. Coefficients and p-values shown in bold italic indicate that a given domain is a statistically significant predictor (p < .05) of the outcome. For 
student characteristics the reference groups were not a Pell Grant recipient, female, not of Northern Marianas descent, and students who had the average 
score for each of the social-emotional competency domains. 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System and Northern Marianas College for first time first-year students at Northern Marianas College during fall 2019 who graduated from one of five 
participating high schools in 2019. 


Table C12. Results of regression model predicting enrollment in developmental math at Northern Marianas 
College 


Enrollment in developmental math 


Predictor variable (ole foMe-hale) p-value 


Student background characteristic 


Pell Grant recipient 3.15 .213 
Gender, male 3.92 .190 
Northern Marianas descent 1.22 .798 


Social-emotional competency domain 


Self-management 2.16 .221 
Growth mindset 0.82 .665 
Self-efficacy 1.63 338 
Sense of belonging 0.27 .149 
Social awareness 2.55 .235 
Constant 1.36 742 


Note: n = 54. For student characteristics the reference groups were not a Pell Grant recipient, female, not of Northern Marianas descent, and students who 
had the average score for each of the social-emotional competency domains. 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System and Northern Marianas College for first time first-year students at Northern Marianas College during fall 2019 who graduated from one of five 
participating high schools in 2019. 


Table C13. Results of regression model predicting enrollment in developmental English at Northern Marianas 
College 


Enrollment in developmental English 


Predictor variable (oJofoMe-hale) p-value 


Student background characteristic 


Pell Grant recipient 1.42 .648 
Gender, male 0.50 315 
Northern Marianas descent 0.30 .065 


Social-emotional competency domain 


Self-management 0.49 .275 
Growth mindset 1.00 996 
Self-efficacy 0.87 .700 
Sense of belonging 1.53 468 
Social awareness 1.46 513 
Constant 1.16 851 


Note: n = 54. For student characteristics the reference groups were not a Pell Grant recipient, female, not of Northern Marianas descent, and students who 
had the average score for each of the social-emotional competency domains. 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System and Northern Marianas College for first time first-year students at Northern Marianas College during fall 2019 who graduated from one of five 
participating high schools in 2019. 


Table C14. Results of regression model predicting completed all attempted credits during first semester at 
Northern Marianas College 


Completed all attempted credits 


Predictor variable (ofoleKma-hile) p-value 


Student background characteristic 


Pell Grant recipient 0.81 .760 
Gender, male 0.55 324 
Northern Marianas descent 1.09 .887 


Social-emotional competency domain 


Self-management 0.77 .637 
Growth mindset 1.13 723 
Self-efficacy 1.24 535 
Sense of belonging 0.94 .912 
Social awareness 0.31 .057 
Constant 5.55 .012 


Note: n = 82. Odds ratios and p-values shown in bold italic indicate that a given domain is a statistically significant predictor (p < .05) of the outcome. For 
student characteristics the reference groups were not a Pell Grant recipient, female, not of Northern Marianas descent, and students who had the average 
score for each of the social-emotional competency domains. 

Source: Authors’ analyses of May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School System 
and Northern Marianas College for first time first-year students at Northern Marianas College during fall 2019 who graduated from one of five participating 
high schools in 2019. 


Table C15. Results of regression model predicting persistence into a second semester at Northern Marianas 
College 
Persistence into a second semester 


Predictor variable (oJoloMe-hale) p-value 


Student background characteristic 


Pell Grant recipient 2.29 .237 
Gender, male 0.98 .978 
Northern Marianas descent 0.58 457 


Social-emotional competency domain 


Self-management 0.65 .617 
Growth mindset 1.73 .234 
Self-efficacy 0.69 399 
Sense of belonging 1.12 .857 
Social awareness 0.99 .986 
Constant 4.44 .032 


Note: n = 80. Odds ratios and p-values shown in bold italic indicate that a given domain is a statistically significant predictor (p < .05) of the outcome. For 
student characteristics the reference groups were not a Pell Grant recipient, female, not of Northern Marianas descent, and students who had the average 
score for each of the social-emotional competency domains. 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System and Northern Marianas College for first time first-year students at Northern Marianas College during fall 2019 who graduated from one of five 
participating high schools in 2019. 


Table C16. Predicted means for first semester college grade point average, by growth mindset response 
category 
NYofeir-]ex-danend (earl mxolpalox-inclaremelelearelin First semester college 


and response category grade point average 


Growth mindset 


1 1.82 
2 2.16 
3 2.49 
4 2.83 
5 3.16 


Note: Survey items were rated ona 1-5 Likert scale, with higher scores reflecting greater social-emotional competency. Only growth mindset was a significant 
predictor of college outcomes at Northern Marianas College (see tables C11—C15). The predicted marginal means are calculated from the regression models 
(see appendix B for how the predicted means were calculated). 

Source: Authors’ analyses of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System and Northern Marianas College for first time first-year students at Northern Marianas College during fall 2019 who graduated from one of five 
participating high schools in 2019. 


Appendix D. Supplementary analyses 

This appendix details results of the sensitivity analyses for research question 2 and of the nonresponse analyses 
for the study’s two analytic samples. The sensitivity analyses compared the findings from the main analysis, which 
used the pooled results from five imputed datasets, with analyses run only on students with complete data to test 
the robustness of the study findings to the methods used to handle missing data. The findings from the sensitivity 
analyses for research question 2 were not significantly different from the findings from the main analyses, 
indicating that the findings were not sensitive to the imputation analysis method used in the main analyses. The 
nonresponse analyses compared the study’s research samples with their corresponding student populations to 
identify differences in student characteristics. Several significant differences between the study samples and the 
student populations were found and are noted. 


Detailed results of multiple imputation sensitivity analysis 
This section presents the analyses that examined the extent to which findings related to research question 2 were 
reflective of the imputed data rather than true sample characteristics. The study team used independent sample 
t-tests to determine whether the regression coefficients of the results for analyses using only nonimputed data 
and of the results of analyses that included imputed data varied significantly. An approach outlined by Weaver 
and Wuensch (2013) was used: 
sap) = 61 = Bal 

SBi-Ba 
where df (degrees of freedom) is equal to the sum of both sample sizes minus 4, B; — Bz refers to the difference 
between the regression coefficients, and sg,_g, refers to the pooled variance of the regression coefficients. The 
study team obtained the corresponding p-values using a two-tailed t-distribution. The findings did not reveal any 
significant differences between the imputed and the nonimputed datasets (tables D1—D4). 


Table D1. Results of sensitivity analysis for linear regression model predicting high school grade point average 


Sensitivity analyses Main analyses 
(n = 424) (eo) Comparison 


Sirlaverlae| Sir-laler-lae| 
Predictor variable error lagelg t(853) p-value 


Student background characteristic 


Grade, 12 0.24 0.78 0.35 0.76 -0.10 .92 
Gender, male —4.52*** 0.77 —4.47*** 0.75 -0.05 .96 
Ethnicity, Asian 5.23*** 0.80 5.02*** 0.78 0.19 85 
Ethnicity, other 5.22* 2.11 5.49** 2.07 —-0.09 .93 
High school, Marianas High School —5.49*** 1.52 —4,72*** 1.30 —0.38 70 
Social-emotional competency domain 

Self-management 1.48 0.84 1.46 0.83 0.02 99 
Growth mindset 1.59*** 0.50 1.57*** 0.48 0.03 98 
Self-efficacy 3.7 78"* 0.48 3.66*** 0.48 0.16 .87 
Sense of belonging 0.35 0.72 0.29 0.73 0.06 95 
Social awareness —2.22** 0.79 —1.98* 0.77 —0.22 83 


*p<.05, ** p<.01, *** p< .001. 

Note: The main analyses used imputed data and the sensitivity analyses did not. For student characteristics the reference groups were grade 11 students at 
high schools other than Marianas High School, female, Pacific Islander/Native Hawaiian, and students who had the average score for each of the social- 
emotional competency domains. 

Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and 12 students in one of the five participating high schools in 2018/19. 
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Table D2. Results of sensitivity analysis for linear regression model predicting grade 10 ACT Aspire math scale 
score 


Sensitivity analyses Main analyses 
(n = 366) (aieast=10)) Comparison 


Sir-lalerelae! Sir-larer-lae| 
Predictor variable lacey e lagele t(801) p-value 


Student background characteristic 


Grade, 12 1.22 0.63 1.36* 0.62 0.16 87 
Gender, male -1.35* 0.62 -1.47* 0.62 0.14 .89 
Ethnicity, Asian 1.81** 0.64 1.79** 0.64 0.02 98 
Ethnicity, other 3.06** 1.85 3.31 1.83 -0.10 92 
High school, Marianas High School -1.43 1.16 -1.03 1.01 —0.26 79 
Social-emotional competency domain 

Self-management -0.11 0.67 -0.17 0.67 0.06 95 
Growth mindset 1.17** 0.39 1.22** 0.39 -0.09 93 
Self-efficacy 2.27*** 0.39 2.27*** 0.40 0.00 >.99 
Sense of belonging —0.67 0.59 —0.63 0.60 0.05 96 
Social awareness —0.67 0.62 —0.63 0.61 —0.05 .96 


*ps05,.** ps 01, *** ps 001. 

Note: The main analyses used imputed data and the sensitivity analyses did not. For student characteristics the reference groups were grade 11 students at 
high schools other than Marianas High School, female, Pacific Islander/Native Hawaiian, and students who had the average score for each of the social- 
emotional competency domains. 

Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and 12 students in one of the five participating high schools in 2018/19. 


Table D3. Results of sensitivity analysis for linear regression model predicting grade 10 ACT Aspire reading 
scale score 


Sensitivity analyses Main analyses 
(n = 369) (n = 383) Comparison 


Sr-Tarer-lae| Sirlaler-lae| 
Predictor variable error lagelg t(804) p-value 


Student background characteristic 


Grade, 12 1.22 0.66 1.13 0.65 0.10 .92 
Gender, male —3.05*** 0.65 —3.04*** 0.64 -0.01 99 
Ethnicity, Asian 1.74** 0.67 1.44* 0.66 0.32 75 
Ethnicity, other 2.42 1.78 2.34 1.76 0.03 .97 
High school, Marianas High School —1.25 1.20 —1.28 1.05 0.02 99 
Social-emotional competency domain 

Self-management 0.75 0.70 0.67 0.70 0.08 94 
Growth mindset 1.52*** 0.41 1.61*** 0.41 —0.16 88 
Self-efficacy 1.82*** 0.41 1.75"** 0.41 0.12 90 
Sense of belonging -0.43 0.62 -0.44 0.63 0.01 99 
Social awareness —1.10 0.66 —1.14 0.64 0.04 97 


*p<.05, ** p<.01, *** p< .001. 

Note: The main analyses used imputed data and the sensitivity analyses did not. For student characteristics the reference groups were grade 11 students at 
high schools other than Marianas High School, female, Pacific Islander/Native Hawaiian, and students who had the average score for each of the social- 
emotional competency domains. 

Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and 12 students in one of the five participating high schools in 2018/19. 


Table D4. Results of sensitivity analysis for negative binomial regression model predicting the number of days 
absent from at least one high school course 


Sensitivity analyses Main analyses 
(n =424) (eo eis) Comparison 


Sir-laler-lae| Srlalerlae| 
High school variable lagele error li ts}os)) p-value 


Student background characteristic 


Grade 12 —0.29** 0.09 —0.29** 0.10 0.00 > .99 
Gender male 0.15 0.09 0.10 0.09 0.38 71 
Ethnicity Asian —0.24* 0.10 —0.19 0.10 —0.36 72 
Ethnicity Other —0.26 0.25 0.17 0.26 -0.25 .80 
High school: Marianas High School —0.34* 0.17 0.24 0.16 -0.43 .67 
Social-emotional competency domain 

Self-management —0.17 0.10 —0.17 0.10 0.00 >.99 
Growth mindset —0.02 0.06 —0.02 0.06 0.00 > .99 
Self-efficacy —0.24*** 0.06 —0.24*** 0.06 0.00 > .99 
Sense of belonging 0.05 0.09 0.07 0.09 -0.16 .88 
Social awareness 0.10 0.10 0.12 0.10 -0.15 88 


*p<.05, ** p<.01, *** p< .001. 

Note: The main analyses used imputed data and the sensitivity analyses did not. For student characteristics the reference groups were grade 11 students at 
high schools other than Marianas High School, female, Pacific Islander/Native Hawaiian, and students who had the average score for each of the social- 
emotional competency domains. 

Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and 12 students in one of the five participating high schools in 2018/19. 


Detailed results for nonresponse analyses 

This section provides the detailed results for the nonresponse analyses. Three analyses were conducted. The first 
analysis (table D5) compared the study’s first analytic sample of 439 Commonwealth of the Mariana Islands 
(CNMI) public school grade 11 and 12 students who completed the social-emotional learning survey and provided 
a valid student identification number with all 1,371 grade 11 and 12 students enrolled in CNMI public schools 
during the 2018/19 school year. The results of this analysis indicate that the first analytic sample had a statistically 
significantly higher high school grade point average and number of days absent from at least one course than the 
student population. The sample also had fewer grade 12 and Pacific Islander/Native Hawaiian students than the 
student population and more grade 11 and Asian students. Finally, the sample had higher percentages of students 
who attended Marianas High School and lower percentages of students from Rita H. Inos High School, Kagman 
High School, and Saipan Southern High School than the student population. 


The second analysis (table D6) compared the average social-emotional competencies of the 439 students in the 
first analytic sample with the average social-emotional competencies of all 890 survey respondents. The students 
in the sample had significantly higher self-management scores. 


The third analysis (table D7) compared the college student outcomes and college student characteristics of the 
second analytic sample of 82 students and the population of 1,252 first-time students enrolled in Northern 
Marianas College in fall 2019.1° The students in the second analytic sample had higher rates of enrollment in 
developmental math and developmental English than the population of first-time students. 


10 The population of Northern Marianas College includes graduates of private high schools who were excluded from the study’s sample. 
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Table D5. Results of nonresponse analysis comparing the means and percentages of the first analytic sample 
of grade 11 and 12 students with all grade 11 and 12 students in Commonwealth of the Mariana Islands public 
schools 


Study sample Saute (<Yaimy ele) ol) r-1ulela) 

High school variable (ee eis) (n = 1,371) 
High school outcome Mean Mean 
Grade point average 83.24*** 82.00 
Grade 10 ACT Aspire math scale score 418.91 419.42 
Grade 10 ACT Aspire reading scale score 418.26 418.13 
Number of days absent from at least one course 17.19*** 21.21 
Student background characteristic Percent Percent 
Grade, 12 0.45* 0.52 
Gender, female 0.46 0.48 
Ethnicity, Asian 0.55*** 0.43 
Ethnicity, other 0.04 0.03 
Ethnicity, Pacific Islander 0.41*** 0.55 
Marianas High School 0.88*** 0.48 
Saipan Southern High School 0.04*** 0.19 
Tinian High School 0.06 0.06 
Da’ok Academy 0.01 0.01 
Rita H. Inos High School 0.00*** 0.05 
Kagman High School 0.004*** 0.20 


* p<.05, ** p<.01, *** p< .001. 

Note: For the high school outcomes the differences between the study sample and the population were calculated using t-tests; for high school student 
characteristics the differences were calculated using chi-square tests. 

Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and 12 students in one of five participating high schools in 2018/19. 


Table D6. Results of nonresponse analysis comparing the social-emotional competency score means of the 
first analytic sample of grade 11 and 12 students with all grade 11 and 12 survey respondents 


Study sample All respondents 


High school variable (n = 439) (n = 890) 


Social-emotional competency 


Self-management 4.02° 3.94 

Growth mindset 3.91 3.81 

Self-efficacy 3.16 3.14 

Sense of belonging 3.27 3.27 

Social awareness 3.74 3.69 
*p<.05. 


Note: Differences between the study sample and all respondents was calculated using t-tests. 
Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and 12 students in one of the five participating high schools in 2018/19. 
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Table D7. Results of nonresponse analysis comparing the means and percentages of the second analytic 
sample with all Northern Marianas College first-time students 


Study sample Student population 
(n = 82) (n = 1,252) 


College student variable Mean or percent Mean or percent 


College outcome 


First term grade point average 2.83 2.75 
Enrollment in developmental math course 0.54*** 0.27 
Enrollment in developmental English course 0.28** 0.15 
Earned all attempted credits 0.76 0.73 
Persistence to a second semester 0.83 0.79 


Student background characteristic 


Pell Grant recipient 0.23 0.28 
Gender, female 0.63 0.63 
Northern Marianas descent 0.40 0.42 


* p<.05, ** p<.01, *** p< .001. 

Note: For the first term grade point average the difference between the study sample and the student population was calculated using t-tests, and the 
results are reported as means. For the other college outcomes and for the college student characteristics the differences were calculated using chi-square 
tests, and the results are reported as percentages. 

Source: Authors’ analysis of linked May 2019 survey data and administrative data from the Commonwealth of the Northern Mariana Islands Public School 
System for grade 11 and 12 students in one of the five participating high schools in 2018/19. 
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